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This study developed twelve instructional modules based on the National Assessment of Educational 
Progress for mathematics content and methods courses for preservice elementary and middle school 
teachers and examined their impact on PSTs’ mathematical content knowledge and self-efficacy beliefs 
about teaching mathematics. The modules help preservice teachers: (1) improve their mathematical 
content knowledge, (2) learn how to use effective methods to teach mathematics; and (3) become aware of 
uses of NAEP. Mathematical content knowledge was measured by instruments from the Learning 
Mathematics for Teaching project and mathematics teaching efficacy beliefs were measured by the 
Mathematics Teaching Efficacy Beliefs Instrument or the Yackel Beliefs Survey. Modules were found to 
increase PSTs’ mathematical content knowledge for teaching and improve their teaching efficacy beliefs. 
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National reports (Project Kaleidoscope, 2006; National Academies, 2003, 2008; Ball, Ferrini-Mundy, 
Kilpatrick, Milgram, Schmid, & Schaar, 2005) pointed out the urgent need to improve the quality of 
science, technology, engineering, and mathematics (STEM) education programs. The National Academies 
(2006, cited in PKAL, 2006) advised increasing “America’s talent pool by vastly improving K—12 science 
and mathematics education” (p. 4-2). Such progress rests on improved mathematical education of teachers. 
Morris (2006) stated, “Preservice teachers rarely exit their mathematics teacher education program as 
experts” (p. 471). The question therefore becomes, how do mathematics teacher educators help PSTs better 
develop their expertise, especially during their early careers? It is difficult to address the learning needs of 
preservice teachers (PSTs) due to the large number of concepts, skills, and strategies that must be acquired 
at a high level of competence to teach successfully. This fact must be kept closely in view by faculty of 
mathematics content and methods (MC&M) courses for teachers. We must carefully examine the goals of 
these courses and ask: (1) Are we giving students enough experiences in the areas we expect them to 
master? (2) Do they have sufficient opportunities to consider problems from both students’ and teachers’ 
viewpoints? and (3) Do they have sufficient opportunities to examine both student work and student 
achievement data? 

It is insufficient to discuss problem solving, development of student conceptions, and assessment of 
mathematical learning, in abstraction. Preservice teacher education must have a strong student focus and 
be rooted in authentic classroom data. Novices need specific experiences in how to analyze student work, 
assess student understanding, and in scoring student work with various rubrics. Previous researchers 
(Morris, 2006; Osana, Lacroix, Tucker, & Desrosiers, 2006) reported the benefits of asking PSTs to 
analyze mathematics teaching episodes from real practice. Morris (2006) described a study where PSTs’ 
abilities to analyze videotaped teaching episodes differed markedly on the basis of whether they were told 
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beforehand that a lesson was unsuccessful or whether they had to make this determination independently. 
Morris’s study suggests that what PSTs focus on and attend to, in analyzing teaching and student 
performance, is linked to what experiences and guidance they receive from teacher educators. 

Current efforts to improve mathematics teaching recognize the importance of helping teachers (a) gain 
depth in their mathematical content knowledge, (b) master specialized content-related strategies needed to 
help children learn mathematics, and (c) learn pedagogical and assessment practices to improve the quality 
of teaching and student learning (Hill, Rowan, & Ball, 2005; Ball & Hill, 2004; Ball & Bass, 2000). We 
believe that the analysis of sound mathematical tasks, discussion of explanations for mathematics 
procedures and concepts, careful analysis of student work, and discussion of assessment practices should 
be a focus of preservice mathematics content and methods courses and that these experiences should occur 
across the teacher education program. The North Carolina NAEP project found preservice teachers to be 
especially receptive to NAEP-related instruction that was clearly linked to examples of student work, 
analysis of that work, and examination of student performance data. 


Purpose 


The goal of the North Carolina NAEP: Improving Mathematics Content and Methods Courses project 
was to modify materials from Learning from NAEP: Professional Development for Teachers (Brown & 
Clark, 2006) for use in preservice mathematics content and methods courses, to expand the materials to 
include more recent NAEP assessment results, and to include, and focus on, more mathematics content. 
While NAEP produces a vast amount of data concerning students’ learning and achievement, this wealth 
of data is not always used effectively within preservice teacher education courses to help PSTs become 
aware of what these data show. The project addressed three research questions, two of which are addressed 
in this paper: 


1. How can mathematics content and methods (MC&M) courses for preservice undergraduates and 
graduate students be improved through the integration of NAEP? 

2. How can NAEP-related materials be used in MC&M courses to help beginning teachers see the 
connections among the following areas: (a) teachers’ content knowledge, (b) student 
understanding, (c) classroom assessment practices, (d) analysis of student performance data, and 
(e) use of NAEP data to address issues of equity? 


The results described here explain the ways in which the modules appeared to influence preservice 
teachers’ mathematical content knowledge and how they seemed to influence their mathematics teaching 
efficacy beliefs. Goals and outcomes of the project include: 


1. The improvement of MC&M courses for elementary and middle school PSTs to produce 
teachers knowledgeable about mathematics content and pedagogy and knowledgeable about 
difficulties students have in learning mathematics topics. 

2. The improvement of MC&M courses to produce teachers able to use NAEP and other 
assessment data to consider issues of equity and to modify teaching to address them. 

3. Improving teachers’ knowledge of various assessment strategies including designing and using 
problem solving rubrics. 

4. The development of (a) multimedia materials that illustrate critical mathematics concepts, 
NAEP-related problems, examples of student errors, statistics concerning student achievement 
on NAEP problems, and activity sheets providing guidance for group analysis of this 
information within MC&M courses; and (b) a project website. 

5. Enhancing instruction and communication between institutions within the North Carolina 
Community College system and the University of North Carolina system. 


The project team involved mathematicians and mathematics educators from the following universities 
and community colleges: Appalachian State University, the University of North Carolina Charlotte, the 
University of North Carolina Wilmington, Forsyth Technical Community College, Mayland Community 
College, and Wilkes Community College. The UNC Teacher Preparation Program Effectiveness Report 
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(Henry, Thompson, Bastian, Fortner, Kershaw, Marcus, & Zulli, 2011) concluded that the following 
programs were outperforming their reference group comparisons in these areas: 


Appalachian State University — Elementary Program: elementary mathematics 

University of North Carolina Wilmington — Middle School and Secondary Mathematics Programs: 
middle school and secondary mathematics 

University of North Carolina Charlotte— Middle School Programs: middle school mathematics (Henry 
et al., 2011, p. 11) 


Thus, we were able to integrate, within the project modules, instructional practices that have been deemed 
effective in improving teacher preparation at the participating institutions. 

The project team included two practicing teachers, one from the elementary level (Ms. Anderson) and 
one from middle school (Mr. Schmal). These teachers helped the team link the NAEP assessment results to 
the realities of daily classroom practice through their contributions to the writing teams and through their 
commentaries concerning how NAEP data mirror the types of student work and difficulties that they 
observe in their classrooms. Twelve modules were produced, four for each of these levels: elementary, 
middle school, and community college. The community college modules are directed at undergraduate 
mathematics courses frequented by preservice teachers. The modules employ a variety of instructional 
approaches including: using active learning strategies; conducting analyses of NAEP results; conducting 
analyses of student work; developing understanding of the mathematics contained in NAEP problems; and 
developing awareness of NAEP rubrics and procedures for assessing student work. The elementary 
modules address the areas of: fraction number sense, addition and subtraction, early algebra, and 
geometry. The middle school modules include the topics: proportional reasoning, geometric and spatial 
reasoning, linear growth and rates of change, and data analysis. The community college modules cover 
the topics of: algebra, probability, proportional reasoning, and spatial reasoning. The study results 
suggest that seeing their college mathematics and mathematics education instructors model more inquiry- 
based pedagogical strategies positively influences and broadens PSTs’ vision of effective mathematics 
instruction. 

The modules are flexible allowing for inclusion within different course structures and time allotments. 
Each module contains: 


1. Purpose 
a. Specification of the mathematical concept(s) addressed 
b. Specification of pedagogical approaches 
2. Overview 
a. Module goals 
b. Module activities 
3. Background and context notes 
a. Includes research brief concerning math concept and relevant pedagogical issues 
b. Includes discussion of common student errors based on research 
c. Examines NAEP student performance data in context of relevant research 
4. Preparing to teach the module (Instructor notes not covered elsewhere) 
5. Introductory PowerPoint presentation for the module (to be presented to PSTs) 
a. Specification of the mathematical concept(s) being addressed 
b. Specification of pedagogical issues addressed 
6. Teaching the Module plan for the university/college instructor that provides: 
a. Goals and objectives of the module 
b. Time required for module 
c. What mathematics is addressed and grade band(s) 
d. How NAEP resources will be used 
e. Materials required 
f. NCTM Principles or Process Standards addressed 
g. NAEP Content Strand Emphasized 


Van Zoest, L. R., Lo, J.-J., & Kratky, J. L. (Eds.). (2012). Proceedings of the 34th annual meeting of the North American Chapter of 
the International Group for the Psychology of Mathematics Education. Kalamazoo, MI: Western Michigan University. 


Articles published in the Proceedings are copyrighted by the authors. 


Teacher Education and Knowledge—Preservice: Research Reports 624 


h. Description of class activities (Activities should include samples NAEP problems, student work, 
and performance data) 
i. Directions for conducting class activities 
j. Student activity sheets 
k. Discussion guide 
7. References 


Each module includes recommended readings and Teaching Notes to help instructors implement the 
activities. The instructor can access PowerPoint presentations as well as a Moodle-based course shell 
that includes the instructional materials. Example modules are available on the project website 
http://ncnaep.rcoe.appstate.edu/. Online and face-to-face professional development opportunities are 
planned for 2013-15. 


Method 


Instructional modules based on NAEP data were developed and implemented in mathematics content 
and methods courses aimed at preservice elementary and middle school teachers who were enrolled at 
three universities and two community colleges from fall 2008 to spring 2010. Thus far, roughly 750 PSTs 
have been impacted by the project. External evaluators analyzed three data sources: (1) PSTs’ performance 
on a mathematics knowledge assessment (LMT); (2) PSTs’ responses to mathematics teaching beliefs 
questionnaires, the Mathematics Teaching Efficacy Beliefs Instrument (MTEBI) (Enochs, Smith, & 
Huinker, 2000) or the Yackel Belief Survey (Quillen, 2004); and (3) faculty responses to the NC NAEP 
workshop surveys and their reported use of NC NAEP modules. The sample included PSTs who were 
given a mathematics content knowledge test and a teaching self-efficacy questionnaire at the beginning 
and end of the semester. The project included sections of experimental courses at each university as well as 
control sections that did not use the project modules. The elementary preservice teachers were given the 
LMT: Grades 4-8 Geometry test or the LMT: Elementary School Number Concepts and Operations—2004 
test, dependent upon their specific course enrollment. The middle school preservice teachers were 
administered the LMT: Middle School Number Concepts and Operations test, as were the community 
college students. It is significant to note that the LMT tests do not just measure mathematics content 
knowledge; they measure a teacher’s knowledge of mathematics for teaching. Thus, this data helps us 
learn not only how the PSTs think about mathematics concepts but it also helps us to form a picture of how 
they interpret possible student responses to certain mathematical scenarios. To ensure comparability across 
all data collected, the analysis included only students in each group, (e.g., course specific, higher education 
setting, and grade-level focus) for which evaluators were able to match both the pre-post scores on each 
instrument. After linking students with their pre and post scores, the evaluators conducted a range of 
statistical tests including significance and regular multiple regression tests, as well as item response 
analyses. 


Key Findings 


The results from the NC NAEP Project suggest that the National Assessment of Education Progress 
can inform instructional practice. This report shares findings based on the evaluators’ analysis—using a 
mix of data from several instruments (i.e., Mathematics Teaching Efficacy Beliefs Instrument (MTEBI), 
the Yackel Belief Survey (YBS), and the Learning Mathematics for Teaching instrument (LMT)) used at 
different stages of the project, and with populations of students from 2 to 4-year undergraduates. Three 
main project components evaluated are discussed below: (1) PSTs’ learning in terms of content knowledge 
and mathematics teaching efficacy beliefs; (2) learning by the college /university faculty impacted by the 
project; and (3) the quality of the modules and accompanying materials. 


Impact on Pre-Service Teachers’ Content Knowledge 


Finding #1. Project faculty were differentially effective in increasing PSTs’ mathematics content 
knowledge for teaching as measured with an LMT test. Because positive growth varied strongly across 
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students linked to different project faculty, the results suggest overall positive effects. The difference in 
faculty effectiveness is significant at p £0.01. Over half of the project faculty had students who gained, on 
average, a standard deviation or more in post-test scores. 


Finding #2. All PSTs showed improvement in their knowledge of mathematics, with elementary PSTs 
making the largest gains. Treatment elementary PSTs showed statistically higher gains in their pre-post 
knowledge of mathematics for teaching as compared to a control. 


Elementary Pre-Service Teachers 


The average number of correct items on the baseline LMT test for all pre-service elementary teachers 
was 10, while the average number correct on the post-test was 12.4. There was little variation in the test 
scores between students in the control and treatment groups—the average pre-test score was 10 for both 
groups, and the average post-test scores were 12 and 12.50 respectively. 

The post-LMT test gain for elementary pre-service teachers corresponds roughly to a 2 to 3-item 
increase, per student, in the raw number of correct items. Considering that the assessment was not 
designed exclusively to match the curriculum of all or any particular course, this gain is a promising 
finding. The standard deviation of the pre and post-test scores for students in the treatment group were 
.892 and .626, respectively, making this gain a third standard deviation in size, and statistically significant 
at p< .001. 

Pre-service teachers pre and post-LMT scores in spring 2010 were higher in the number of overall 
correct items by 2 than compared to teacher’s pre-post scores in fall 2009. An analysis of matched items 
from the pre-post LMT showed that spring 2010 students performed significantly better on items designed 
to measure operations content knowledge than other students. Correct responses to items 5 and 8 were 
positively and strongly predictive of teachers’ knowledge of mathematics, and higher raw scores than other 
matched items. These results suggest that as the project proceeded and the project modules were more 
effectively integrated within the courses, the benefits became more pronounced. 


Middle School Pre-Service Teachers 


The number of middle schools pre-service teachers for which there were matched pre-post test scores 
for was considerably smaller than the sets for the community college and elementary pre-service teachers. 
This data is reported for comparison purposes, but we suggest the results be viewed with caution because 
of the relatively high standard error compared to other sub groups. Analysis of the pre-post LMT scores 
for pre-service middle school teachers revealed a statistically significant increase in post-test scores at the 
5 percent level overall, and in both fall 2009 and spring 2010. 


Finding #3. Preliminary results indicate a high degree of correlation r = 0.78 between PSTs’ personal 
beliefs of math and math instruction and their performance on the LMT. The relationship between 
knowledge of mathematics teaching was strongest with the personal efficacy sub-scale. 


Impact on Pre-Service Teachers’ Self-Efficacy 


Finding #4. Evidence suggests that the NC NAEP modules used in different contexts influence PSTs 
self-awareness and confidence in their personal efficacy for mathematics and mathematics instruction. We 
examined the relative effects of instructional environment (2 or 4-year university/college) and instructor 
with different characteristics on PST outcomes. Four unique pair-wise comparisons of PST beliefs were 
conducted: (a) 4-year university PSTs relative to community college PSTs, (b) elementary school PSTs 
relative to middle school PSTs; (c) elementary pre-service treatment teachers relative to elementary pre- 
service control teachers; and (d) fall 2009 PSTs relative to spring 2010 PSTs. 

Among the aggregate results on the MTEBI, PSTs responses indicate changes in nearly every personal 
efficacy item, with significantly positive changes (p < 0.001) on three items: (e.g., “I know how to teach 
mathematics concepts effectively’). In nearly every dimension (e.g., subject matter-knowledge, pedagogy 
and subject-specific pedagogy), PSTs from spring 2010 had markedly higher self-efficacy beliefs toward 
mathematics, with elementary PSTs showing dramatic shifts—both experimental and control. Both 
elementary pre-service and middle school PSTs from 4-year universities, showed a statistically significant 
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differential (p £ 0.001 and p < 0.05, respectively) intra-group in their post-mathematics efficacy scores— 
indicating changes in their mathematics content and PCK. 


Finding #5. At the end of the semester, PSTs, in general, noted changes in their attitudes that they can 
positively influence student learning—their belief in outcomes expectancy. Elementary and middle school 
PSTs from spring 2010 showed significant changes (p < 0.05) in their outcomes expectancy beliefs when 
compared to candidates from Fall 2009—both experimental and control. 


Impacts on Faculty Instructional Practice 


Finding #6. Project faculty found value in using project resources to improve their own instructional 
effectiveness. NC NAEP modules forged connections between mathematics content, instructional practice 
and assessment, to help faculty better prepare PSTs. 


Finding #7. Non-project faculty found the NC NAEP materials very useful. They reported that the 
training and materials helped them form a personal action plan for using NC NAEP resources and illustrate 
concrete resources and strategies for improving students’ preparedness. 


Finding #8. NC NAEP resources helped to challenge participating faculty’s preconceptions concerning 
their teaching of mathematics content and methods. Participating faculty recognized the importance of 
changing their practice to evaluate their students’ preparedness and to emulate authentic classroom 
activities. 


Conclusions 


The NC NAEP modules had important effects on the development of PSTs’ mathematical content 
knowledge, and improved their personal efficacy, outcomes expectancy, and attitudes toward mathematics. 
The modules provided a meaningful format from which to draw situated, authentic resources to support 
critical thinking and reflection about mathematics instruction. These modules have the potential to play a 
critical role in the preparation of high-quality, well-prepared teachers of mathematics. However, some 
limitations of the evaluation should be mentioned. While the LMT is a reliable measure of mathematical 
content knowledge for in-service teachers, its use with PSTs is preliminary (Gleason, 2010). To address 
this concern, we plan for future evaluations of the modules to validate individual LMT results using either 
an assessment of PSTs use of mathematics content during situated classroom teaching or using interviews 
where students explain their thinking and solution process to the test items. Gleason also suggests that the 
reliability of the LMT with preservice teachers is strengthened through the use of multiple instruments. 

In addition to the findings discussed above, the project team submitted the modules to an external 
mathematics educator from the Pennsylvania State University for evaluation. The evaluator provided 
detailed feedback concerning the quality and utility of the modules, noting: 


Overall, I can see that the project has worked hard to integrate NAEP items, results, and student work 
into modules that are intended to be disseminated for use in PST education. Developing meaningful 
activities for PSTs is challenging; developing accompanying facilitator notes adds levels of difficulty 
to the task. The developers of the materials are to be commended for their work thus far, particularly in 
the area of integrating NAEP into your materials. 


Some of the evaluator’s suggested revisions include: (a) limiting the scope of some of the modules, 
particularly the community college materials, to make them easier for instructors to implement; 
(b) addressing the issues associated with making referenced video clips more accessible; and (c) changing 
some of the facilitator notes to provide more guidance to instructors concerning how to best implement the 
modules. We are making revisions to the modules based on this helpful feedback. 

Another indicator of the quality of the project modules is their adoption for use in selected courses 
within a new multi-university program in North Carolina for an Add-on Certificate in Elementary 
Mathematics (which is similar to a K—5 mathematics specialist). Evaluation of the pilot add-on certificate 
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program showed improvements in mathematics content knowledge of the teachers enrolled as compared to 
a control group. 

This project provides an example of how different communities of professionals can contribute to the 
effectiveness of mathematics teacher preparation programs. Mathematicians and mathematics educators, 
including instructors from various levels of higher education (doctoral institutions, 4-year institutions, and 
community colleges), and classroom teachers all have a crucial role to play in the development and 
implementation of authentic instructional resources in mathematics teacher education. Such cooperation 
among professionals and institutions facilitates transitions across the continuum, including the transition 
that students make as they progress through a teacher education program and emerge as an effective 
beginning mathematics teacher. 
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